Contextualized embeddings for semantic change detection: Lessons learned

We present a qualitative analysis of the (potentially erroneous) outputs contextualized embedding-based methods for detecting diachronic semantic change. First, we introduce an ensemble method outperforming previously described approaches. This is used as basis in-depth degrees change predicted English words across 5 decades. Our findings show that can often predict high scores which are not undergoing any real shift in lexicographic sense term (or at least status these shifts questionable). Such challenging cases discussed detail with examples, and their linguistic categorization proposed. conclusion pre-trained language models prone to confound changes senses contextual variance, naturally stem from distributional nature, but different types issues observed based on static embeddings. Additionally, they merge together syntactic aspects lexical entities. propose range possible future solutions issues.